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SPECIFICITY OF INFORMATION IK DATA-BASED 
DECISION MAKING IN SCHOOLS 



Leigh Burstein 
University of California, Los Angeles 



Making decisions is like speaking prose— people 
do it all the time, knowingly or unknowingly.... 
The study of decisions addresses both normative 
and descriptive questions. The normative analysis 
is concerned with rationality and the logic of 
decision making. The descriptive analysis, in 
contrast, is concerned with people's beliefs and 
preferences as they are, not as they should be. 
The tension between normative and descriptive 
considerations characterizes much of the ;tudy 
of judgment and choice. (Kahneman and Tversky, 
1984, p. 341) 



The above quotation, taken from the 1983 American 
Psychological Association Award address by Amos Tversky, carries 
a message with multiple meanings that reverberate throughout the 
latest round of efforts to use information efficiently and 
effectively to improve educational quality. The message is that 
as in other areas of human endeavor, the benefits of data-based 
decision making in schools are not inherent but derived from *he 
circumstances in which it is attempted. On the one hand, removed 
from the context of the decision, the quality of the data 
available for decision-making can be normatively established 
through generally agreed-upon technical standards. On the other 
hand, someone or some group makes the decision. Thus in any 
specific instance, the human beliefs and preferences of the 
decision-maker (s) impose their own meaning and "context" on the 
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data upon which the decision is intended to be based. The 
picture is further ccr. plicated (muddled?) in that the human 
beliefs and preferences guiding individual action are likely to 
be influenced by the social and historical context, i.e., by the 
societal, political and organizational circumstances prevailing 
at the time when the decision must be made. 

This analysis of the factors influencing human decision 
making, and its implications for the judgments and choices 
("decisions") that result from juxtaposition of data with 
decision context, serve as a backdrop for the consideration of 
selected issues regarding the specificity of information in data- 
based decision making in schools. The term "specificity of 
information" refers to the units of information and their degree 
of divisability or decomposability. Specificity is a relative 
attribute. In terms of data collection and reporting in 
educational practice, the most specific piece of information 
(data) is typically the response of an individual to a single 
question (test question, survey item, demographic question) or 
the classification of a higher organizational unit (classroom, 
school, district, state, country) on a single attribute (e.g., 
remedial class, private school, Southeastern state, 
industrialized country) . Summarizations (counts, averages, 
totals, etc.) across persons for a given question or across 
questions for a given person, or both, yield less specific data. 

Different summarizations over a person ( organizational unit)- 
attribute array of data result in different degrees of 
specificity of information. The question then is how does the 
choice among alternative summarizations , and hence the 
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specificity of information, interact with decision contexts to 
influence data-based decision making in schools . In our 
examination of this question, we draw upon work from efforts to 
improve the quality and utility of information use for 
educational decision-making through the creation of instructional 
information systems in schools (Bank & Williams, 1983a, 1983b, 
1985; Burstein and Sirotnik, 1984; Cooley & Bickel, 1985; 
Hathaway, 1984; Idstein, 1984; Sirotnik, Burstein & Thomas, 
1983). In the instructional information systems (IIS) movement, 
the increasing availability of amenable technology (the hardware, 
software, and human resources for educational uses of computing) 
has made it possible for school districts throughout the country 
to develop comprehensive information systems to inform 
educational decision making at various levels within the 
districts. 

The remainder of the paper will proceed as follows. In the 
next section, our reason for focussing on the specificity of 
information point in light of the variety of issues and concerns 
that have arisen during the course of the Systemic Evaluation 
(SE) Project is provided. We then consider selected 
issues that in theory should influence the choice of appropriate 
specificity of information in data-based decision making in 
schools. The points raised during the general analysis in 
Section 3 are then applied to the work on instructional 
information systems (section 4). In this section, examples from 
existing practice are used to illustrate the specificity choices 
that actually are made and what these choices imply about the 
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features of the decision context. The paper concludes with 
suggestions intended to advance the state of pra< tice with regard 
to balancing the competing factors that impinge :he choice of 
specificity in data-based decision making in schools. 

Intellectual Antecedents of the Pf per 

From its inception the SE Project has been concerned with 
the use of comprehensive information from multiple sources 
(students, teacher, s local school and district administrators, 
parents, community members) about the context, processes, and 
practices in local school settings that can be used in efforts to 
improve the quality of schooling (Burstein and Sirotnik, 1982; 
1984). This concern reflected the integration of two highly 
interrelated conceptual perspectives (contextual appraisal and 
multilevel evaluation design and analysis), one derived from 
experiences in studying schooling and informed by a specific 
ideology regarding educational change (Sirotnik) and the other 
derived from perceived methodological inadequacies of most large- 
scale research and evaluation efforts focussing on educational 
(school, classroom, teacher, program) effects (Burstein). 

The primary focus of the SE Project itself was on the 
contents of comprehensive information systems, their uses at the 
student, classroom, school, and district levels; and the 
mechanisms that enhance the likelihood that such systems would 
become ongoing and enduring components of the school improvement- 
process. In contrast to the complementary CSE work of the 
Management of Instructional Information Systems Project (Bank and 
Williams, 1984), the SE work emphasized the nature, technical 
organization, and presentation of the information itself as well 
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as the 'it between the information and information delivery with 
the needs of diverse school personnel. 

Despite its intended thrust and emphasis, a reading of the 
most recent set of SE project reports (Dorr-Bremme, 1985; 
Sirotnik, Dorr-Bremme, fc Burstein, 1985; Sirotnik & Burstein, 
1985) clearly depicts the impact of human and environmental 
conditions (interpersonal, social, organizational, political) on 
information selection and use. In fact, one might reasonably 
conclude that the impact of these conditions exceeds that of the 
technical and methodological features of competing alternatives 
at virtually every step along the way in developing, 
implementing, and using the information system in our case study 
school. 

This result is consistent with findings from Bank and 
Williams' research and from the papers and discussion at the 
February 1985 CSE conference on Information Systems and School 
Improvement: Inventing the Future. From Bank's socio- 
organizational perspective (also reflected heavily in Dorr- 
Bremme, 1985), this phenomenon might be construed alternatively 
as a) a natural consequence in an environment (clearly 
characteristic cf our case study site) where the debate between 
"teaching as a craft/schools as organic communities" and 
"teaching as technology/schools as bureaucratic institutions" 
rages on; b) as an instance where the compatibility and 
incompatibility of assumptions and values underlying education on 
the one hand and management information systems on the other are 
evident; and c) as a familiar reaction of an educational 
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organisation to the change process itself with the information 
system serving as the technological innovation. 

While these socio-organizational explanations for the 
relative importance of human and environmental conditions are 
both compelling and appealing, they do not exhaust the 
possibilities that can account for the results from our case 
study. Moreover, these socio-organizational explanations do not 
lead necessarily to a system design in other organizational 
settings that is viable for the decision contexts in which they 
are introduced. Perhaps basing the design of information systems 
on principles derived from the psychology of human decision 
making in context is a more functional strategy, leading to an 
improved fit between the inherent qualities of information and 
its value and utility within its desired decision context. 

This present paper then is in response to curiosity about 
whether ideas developed from the literature on the psychology of 
human decision making might warrant greater consideration in the 
current attempts at data-based decision making in schools. The 
focus on the presumably technical/methodological choices 
regarding the specificity of information hopefully accomplishes 
its intent of grounding the theoretical arguments within the 
practical school settings in which decisions are intended to be 
made. 

Choosing the Appropriate Specificity of Information 
As was pointed out at the beginning of the paper, multiple 
factors can impinge on the judgments of the quality of data for 
decision making. Normatively established ceteris parabus 
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technical considerations must be balanced against the human and 
environmental circumstances that inhere in the decision context. 
The extensive literature on the psychology of human decision 
making (The work of Tver sky and Kahneman (e.g., Kahneman & 
Tversky, 1979; 1984; Kahneman, Slovic, & Tversky, 1982; Tversky 
Kahneman, 1981) is the most widely cited and acclaimed although 
the topic of decision making is shared by many disciplines.) 
offer possible explanations for choices that are likely to be 
made in such circumstances. 

The following simple heuristic model for data-based decision 
making may help to focus the discussion. Let CASI denote the 
decision of interest, in this case the Choice of Appropriate 
Specificity of Information. If ITQI denotes the Inherent 
Technical Quality of the Information, CDM denotes the 
characteristics of the decision-maker ( s ) , and EC denotes the 
environmental circumstances component of the decision context, 
then the decision can be modeled as follows* 

CASI = f(ITQI X CDM X EC) (1) 

(In the equation, the symbol x is used to specify that the 
relationship may be interactive rather than simply additive.) 

Each of the factors influencing the decision has its own 
potentially multidimensional composite of attributes that 
accounts for its salience with regard to the decision of 
interest. Several of the major attributes of these factors with 
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respect to the specificity of information within a data-based 
decision making in schools context will be discussed next. 
Inherent Technical Quality of Information 

The question of the appropriate specificity of information 
is governed in part by inherent features of the technical quality 
of information. Holding other factors constant, we can say the 
following with regard to the relationship between the technical 
quality of information and its specificity: 

1. Assuming perfect measurement and infinite capacity to 
assimilate information, the technical quality of information is 
directly related to its specificity. Thus the full person- 
attribute array of information is of technically higher quality 
because it retains more of the information than any other choice 
of degree of specificity. According to this standard, finer 
content distinctions (e,g, subskills rather than skills) on 
achievement tests; the responses to single items from surveys 
rather than the scales composed from the items; and the scores of 
individuals as opposed to the aggregated scores for the groups 
(classes, schools, etc.) to which they belong are of higher 
quality than their alternative. These conclusions held regardless 
of the underlying dimensionality of the responses. 

2. The assumption of infinite capacity to assimilate 
information is practically untenable. At some point the amount 
of information to process exceeds an individual's capacity to 
process it and employ it effectively. While this tipping point 
might vary across individuals (a point relevant to the discussion 
of decision maker characteristics later on) , once the available 
information exceeds the tipping point, the quality (in terms of 
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its value to improving judgment and choice) of the additional 
information is minimal at best. At worst the information overload 
that results inhibits judgment; in such instances the quality of 
the complete set of available information deteriorates. The 
implication of this phenomenon for the choice of specificity of 
information is that at some point, information reported in a less 
specific form (through summarization or selective reporting) may 
be of higher quality than the complete data array. One could 
argue that this provides the substantive rationale for interest 
in the fields of statistics and data analysis. 

3. The assumption of perfect measurement in obtaining 
information is also likely to be untenable. Human responses to 
stimuli (questions) are likely to vary depending on the 
conditions under which the responses are elicited. Tnose 
conditions include features of the occasion (time of day, period 
in time, physical netting, organizational setting, etc.) and of 
the stimuli (content, type, wording, format, and location of the 
question among others). Variability of response (either 
systematic or random) on a condition that is irrelevant to the 
decision of interest reduces the quality of the information with 
respect to that decision. Summarization over responses 
associated with such conditions improves information quality by 
reducing the noise associated with variation on the condition. 
Here, again, less specific information can be of higher quality 
than more specific. As in point 2, concerns for the impact of 
relaxing a basic assumption about information quality leads us 
directly to a field of quantitative methodology (in this case 
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psychometric* with its basic concepts for the validity, 
reliability and scaling of measurments ) . Summarizations that 
improve the validity, reliability, and scale properties of 
information for a a4yej! decision lead to higher quality 
information- 

While other factors might be mentioned, the three already 
cited (degree of specif icicy (SP), mass or volume of information 
(IM) and psychometric quality (PQ)) largely determine the 
inherent technical quality of information (ITQI). These factors 
operate interactively and in the case of information mass 
nonlinear ly. Accordingly, the relationship can be written as 
follows: 

ITQI ■ f(SP x IM x PQ) (2) 

Characteristics of the Decision Maker(s) 

A complete delineation of the characteristics of the 
decision maker that might influence their response to information 
will not be attempted. With respect to judgments about the 
appropriate choice for specificity of information, the most 
salient characteristics of the decision maker wruld appear to be 
the knowledge base with respect to both the decision of interest 
and the technical qualities of data, the ascriptive features of 
their role in the educational setting, and values and 
preferences with respect to both their role and to the utility of 
information in educational decision making. We discuss each of 
these sets of characteristics briefly below. 
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Knowledge Base 

The decision maker brings to the decision contexts an array 
of understandings about the decisions to be made and the intended 
bases for the decision. One subset of this knowledge base has to 
do with the decision of interest. Stated simply, a naive decision 
maker responds differently from an experienced decision maker. A 
teacher who has never taught a specific lesson or a specific 
class of students responds differently from another with 
excensive practice or familiarity with the material or class. 

Individual differences in the decision maker's understanding 
of the technical properties of information are also important. 
In most school settings there is likely to be substantial 
variation in understanding of statistical and psychometric 
concepts among teachers and administrators. Given the dependence 
of most information system applications on some common body of 
knowledge with respect to the properties of tests and 
questionnaires (not to mention the instructional, curricular, and 
psychological theory underpinning the choice of information to 
collect and report), it is obvious that this source of decision 
maker knowledge is likely to have a significant impact. 

We decided not to include e -eparate "dimension" to 
represent individual differer .ong decision makers in their 

capacity to ass* filiate information (see earlier discussion about 
the effects of what we have termed mass of information). To some 
degree ».nis capacity is learned through training and experience. 
Regardless of its source, this attribute is hard to disentangle 
from other characteristics of the decision maker knowledge base 
in practice. 

11 
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Role in Educational Setting 

The point of mentioning this characteristic is to simply 
highlight the fact that the roles and job responsibilities of the 
decision maker within the organization influence the reaction to 
information. Teachers' information needs are different from 
administrators because of the differences in job responsibilities 
and hence decisions that must be made based on data. The same 
could be saiu for other professional and lay categories of 
interested parties in educational settings (e.g., counselors, 
parents, school board members, students). 

Roles within a category also vary in responsibilities as 
well. For instance, teacher responsibilities are typically quite 
different for primary schools with their intact classrooms versus 
secondary settings with multiple sections of varied subject 
matter content. Likewise the role of the school administrator 
will also change from primary to secondary schools. (This point 
was discussed in greater detail in an early project report 
(Burstein, 1983) and served as one of the main justifications for 
the choice of a secondary school focus of the SE Project work on 
information systems.) 
Values and Preferences 

The American penchant for individuality and autonomy in 
thought and action is very evident in the diverse ways in which 
school personnel approach their educational responsibilities. 
Orientations toward education in general, beliefs about the 
purpose of schooling (e.g., relative importance of academic, 
social, personal, and vocational development of students), views 



9 

ERIC 



12 

15 



The above litany about diversity of beliefs and preferences 
of school personnel in our educational system pinpoints the 
dilemma faced by designers of information systems for data-based 
decison making in schools. For which type of personnel should 
the system be designed? Must there be a critical mass of 
personnel within a setting with commonality of purpose, beliefs and 
preferences to warrant devoting resources to developing 
information systems? Many of the contributions to Bank and 
Williams' book (1985) deal with variations on this topic; it is 
also a major theme of Dorr-Bremme's analysis the reasons why our 
case study worked as it did. Certainly there are instances where 
the complications associated with diversity have been manageable 
(e.g., Cooley & Bickel's work), but the difficulties of creating 
an '• iformation system responsive to the diversity in schools 
remains a major obstacle to such efforts. 

Environmental Circumstances 

The extra-individual and extra-data circumstances of time 
and place clearly must be figured into any equation purporting to 
account for the appropriate choice of specificity of information 
for data-based decision making in schools. Virtually all of the 
accounts of efforts to develop such systems (e.g., Bank & 
Williams, 1983; Coleman £ Karweit, 1972; Cooley & Bickel, 1985 in 
addition to our own work) vividly portray the impact of social, 
organizational, political, and historical conditions, both 
internal and external to the setting, on their development, 
implementation, and ultimate use. 
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Within our own study, the influence of these environmental 
conditions are considered extensively in Dorr-Breraroe • s (1985) 
discussion of the social organization of the school (scarcity of 
time and limited communication, multiple agendas in the school, 
distribution of power and division of roles, the district contex) 
and of leadership and support. Obviously, the presumably distinct 
perspectives (social oiganization versus the psychology of human 
decision making) begin to merge at this point even though a 
social organizational viewpoint concentrates on the details of 
the environmental conditions themselves while the human decision 
making viewpoint reflects outward from the decision maker upon 
those features of the environment that are likely to influence 
behavior. 

Two additional points warrant special mention in the 
discussion of environmental circumstances. First, there is a 
clear need to attend to compositional influences as an 
environmental factor of import for decisions about information 
specificity. The characteristics of other decision makers within 
the setting (other teachers, administrators, etc.) establish a 
common or diverse peer culture within which an individual 
decision maker must function. An individual's actions (in this 
case, responsiveness or resistance to information) are likely 
differ if she or he is placed in a setting composed of colleagues 
with a different profile of characteristics. 

Second, it is hard to over-emphasize the influence broader 
societal circumstances operating at the given moment in time have 
on the decision making process. Schools go through historical 
periods where the impetus for action is internally motivated, 

15 



18 



driven by self-examination mixed with opportunity (in the form 
of resources (intellectual, psychological, economic) to support 
change efforts). At other times, the motivations for actions are 
more externally derived; national concerns about the condition of 
education, calls for educational accountability, and reforms 
originating from the state and federal level are clear instances 
of the latter. 

These types of external pressures cast a different pallor 
over efforts to employ information systems in school improvement. 
Most of the discussions about the value of ownership of ideas get 
lost to the pressures to react to external barometers of well-being. 
One often finds an entirely different set of criteria employed to 
develop systems in response to these external pressures. It is 
also reasonable to expect that a different set of decision maker 
characteristics (e.g., survival skills) are likely to be salient 
in the decision making process, and consequently, in the choice of 
appropriate specificity of information. 

Examples from Instructional Information Systems Work 
Above we have raised the question of whether a psychology of 
human decision making perspective might be applicable to the 
choice of appropriate specificity of information in data-based 
decision making in schools. We then proceeded to delineate the 
components such a model would entail. In this section we 
illustrate how both the model and its components can be used to 
interpret the selection of degree of specificity of information 
in several instructional information systems. Our illustrations 
are necessarily selective as the combinations of information 




features with decision maker character sties and environmental 
conditions expands quite rapidly. Moreover, most of the examples 
will be drawn from our own work and from school districts 
participating in Bank and Williams' projects. (We will retain the 
practice from our earlier reports of maintaining the anonymity of 
the districts although che examples were taken from non- 
copyrighted documents appearing in the public domain. ) 
Example 1 zz. Instructional Mana ge ment Syster. 

The first set of examples are sample reports from an 
instructional management system (IMS) in operation a medium size 
district on the East coast. The system was designed to among 
other things) : 

1. implement district-wide measures of student progress through 
the basic skills curriculum 

2. enable teachers to determine student academic status 
throughout the school year (including articulation from 
grade-to-grade ) 

3. provide teachers and administrators with targeted 
information for efficiently evaluating student progress and 
instructional programs 

4. establish an unambiguous basis for communicating student 
progress to parents. 

The system description indicates that 2600 tests measuring 1300 

objectives (over all grades) in the basic skills are contained 

in the system. A student takes an IMS test when the teacher 

think*, the student has mastered a skill. The teacher gets a 

report back immediately and student records at the district level 

are updated daily. Parents receive reports periodically. 

Data Displays Exhibits 1 through 3 are reasonably typical of the 

reports generated by instructional management systems. These 
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reports axe targeted to the classroom teacher (a parent version 
of Exhibit 1 provided a list of objectives mastered and not 
mastered with each objective described by a brief statement (e.g. 
"write a ratio for a word problem using ratios")). 

the exhibits differ in degree of specificity of information 
and in the types of decisions they are intended to inform. 
Exhibit 1 is a virtually complete record of student status with 
respect to the objectives in grade 8 mathematics. This sheet is 
presumably for the teacher's file to assist in parent 
conferencing and keeping track of w' t a given student has to 
complete in the math curriculum. Tne report is virtually devoid 
of numbers (not even a running tibulation of total objectives 
mastered) and there is nothing to suggest any concern for the 
psychometric basis for the student master/non-master 
classification on a given objective. Moreover, all objectives are 
given equal billing. 

what does Exhibit 1 suggest about what designers presume 
about the characteristics of teachers? First, the sheer number 
of objectives (31 objectives from 11 skill areas) presented with 
limited elaboration suggests that the teachers are presumed to be 
competent subject-matter specialists capable c»t fine-grained 
distinctions in their teaching. Second, the absense of 
numerical information and statistical and psychometric 
summarizations would seem to indicate that teachers are either 
not expected to have significant quantitative or assessment 
expertise or that this report is simply to serve a more limited 
purpose (see earlier comments). Basically, the decisions Exibit 
1 could inform appear to be limited to those dealing with a 
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(A) + 






NON -MASTERY 


2 


04/10/84 


(A) - 


06/05/84 


(B) - 


MASTERY 


1 


05/01/84 


(A) + 






mast:ry 


1 


04/10/84 


(A) + 






MASTERY 


1 


05/14/84 


(A) + 






MASTERY 


2 


05/01/84 


(A) - 


06/05/84 


(B) + 


NON -MASTERY 


2 


05/01/84 


(A) - 


06/05/84 


(B) - 


NON -MASTERY 


2 


05/01/84 


(A) - 


06/05/84 


(B) - 


MASTERY 


2 


05/14/84 


(A) - 


06/05/84 


(B) + 


MASTERY 


2 


05/14/84 


(A) - 


06/05/84 


(B) + 


MASTERY 


2 


04/10/84 


(A) - 


06/05/84 


(B) + 


Lit}, & Craphing 






MASTERY 1 


05/01/84 


(A) + 






MASTERY 


2 


05/14/84 


(A) - 


06/05/84 


(B) + 
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specific student's progress and are more for description than for 
evaluation. 

Exhibit 2 contains a student-by-objective grid for the class 
as a whole. There is again a substantial amount of specificity 
and limited summarization. The codes for reporting status are 
again non-numeric. This report is presumably intended to 
provide a ready reference for the teacher of objectives that the 
class, or a group, have yet to master (working down a column) or 
students who have a lot of objectives to be completed (working 
across a row) . This particular report was generated early in the 
year so one might presume that general content selection 
decisions and any decisions about instructional grouping might be 
informed by this information display. There is, again, a strong 
indication that the teacher, as the decision maker designated to 
use these data, can tolerate a substantial amount of specificity 
if it's curricularly and instructionally targeted and not 
heavily numerical. Also, psychometric properties of the tests 
themselves or the reporting categories are unimportant. 

The specificity in Exhibit 3 is again substantial but 
focussed on a different type of decision that the teacher might 
make. Here the instructional objectives is the focus and 
student's actual item response patterns on the test of the 
objective are reported. The feedback loop is short term here as 
this information is intended to inform the teacher about the 
kinds of misunderstandings that remain after instruction on the 
objective. For example, note that many students (15 of 32) chose 
alternative C for item 2. li this item taps an aspect of the 
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Exhibit 2 



INSTRUCTIONAL MANAGEMENT SYSTEM CIMS GROUP MATRIX REPORT 

MIDDLE SCHOOL CRITICAL OBJECTIVES 



M (Math) 

0 ( 

( Grade 8 
6 ( 

(Strands) ADDDDDDDDDDFFFGGGGMMMNNNNNNNSSS 
DCCCCCCCCCVRRRMMMMLLSMTTTTTTBTT 
(Objectives) 0000000111000000010000000000000 
1123456012212736811211156789113 



Jones, John ????????? ??????? ? A 7A?? 

Doe, Susan XXXXXBA XA7X XAA X? BB XXBB7 B 

Smith, Joseph X XX AB AXA A XX X 

Johnson, Thomas XX X X X XX B X XX 

Brown, Alex 777777777 7777777 7 A A 77 



********************************************** 

* * 

* * 

* 7 - no test data * 

* (blank) ■ mastery * 

* A ■ non-mastery (Form A) * 

* B ■ non-mastery (Form B) * 

* X ■ non-mastery (Forms A & B) * 

* T ■ Teacher certified mastery * 

* a « Teacher certified non-mastery A * 

* b - Teacher certified non-msstery B * 

* * 
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Exhibit 3 



INSTRUCTIONAL MANACEMKNT S YSTliM 
SUMMARY RBPORT BY OBJECTIVE 



OBJECTIVE CODE • V05GRO1 

S«!ILI. PM " ,t W , Uh nouns "^erlined and choices of 
pronouns, correctly substitute pronouns for nouns. 



NON-MASTERY 
STUDENT 

264503 LOA SHE* OON 
4 57 358 'OX WIOx • L 
671827 . *ORt RYAN ' 
803346 C* RTN. ROB. K 

TOTAL NON-MASTERY (Count » 4): 



FORM 


DATE 


CORRECT 


A 


U/17/83 


2 


A 


U/17/83 


3 


i. 


11/17/83 


3 


A 


11/17/83 


3 



ITEM S 

(AC++A) 
(+C+A+) 
(+C++A) 
(+A++A) 



010923 

011018 

011043 

01 1090 

011115 

01 1370 

01 1391 

011410 

011415 

011421 

01 1422 

01 1437 

025213 

128528 

154764 

207147 

2072 7 2 

238749 

369296 

562525 

570828 

573861 

574064 

584967 

649624 

702145 

826440 

839335 

TOERLC 



MASTERY 
STUDENT 

*ACKl <J CYl *IA 

UBAi \ CRk R 
* 'EGA* 'EATH. D 
SA. DEV M I 
VAL. NT C. ISTIN * 
CEHRK MS Bx T TANY 
" *>NACh » REV • L 
L 'VELA. » PAT, CIA 
2L JEWS. * BROC *. A 
MIN KEV * J 
PRESk STE. NIE D 
'.^BSTt BROv \ E 
1 'NN M. HAEL ' 
S-. f ER Fi OERI. M 
SHi ' JOHl, ".HRU npHER 
SON'x X JON *HAN 
CAROJ. ■» JUL A 

1EV00K JOSE* U 
C T NN HE. m £ 
Wt "*E WILi. M A 
WAL MEATKt K 
KAUFi. 0 JIL. \ 
EMERSO. TAMIE 

•STEN i 'RIFAh ' 
Cw -INCS RISTI#. 
CAV.. ' BRI. 
MALOY T CHEL I. 

. RX/l». ' J 
MASTERY (Count • 28): 



FORM DATE 

A 11/17/83 

A U/17/83 

A 11/17/83 

A 11/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/l"/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 

A U/17/83 
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CORRECT 

5 

4 

5 

4 

4 

4 

4 

4 

5 

4 

5 

5 

5 

4 

5 

5 

5 

4 

4 

5 

4 

5 

5 

5 

4 

4 

4 



ITEMS 

(+++++) 

(++C++) 

(+++++) 

(+C+++) 

(+C+++) 

(+C+++) 

(+C+++) 

(+C+++) 

(+++++) 

(+C+++) 

( +++++) 

(+++++) 

(+++++) 

(+C+++) 

( +++++) 

<+++++) 
(•M-+++) 
(+C+++) 
(++++A) 
(+++++) 
(++++A) 
(+++++) 
(+++++) 
(+4++0 
(+C+++) 
(+C+++) 
(+C+++) 

(+++++) 



objective that was covered by the teacher, perhaps some targeted 
review is in order. 

Implicit Model of Decision Making. Stepping back from the 
separate exhibits to consider the set as a whole, we obtain a 
reasonably consistent profile of the designers' implicit model for 
teacher decision making about instruction and its impact on the 
choice of specificity of information. With respect to the inherent 
qualities of the information (ITQI), instructional decisions 
seem to require very detailed information targeted to individual 
students (high SP). When so targeted, teachers can assimilate a 
large amour.t of information (high IM). Also psychometric 
properties of information are not very important (low concern for 
PQ) ; presumably, the opportunity for remediation and re- 
examination in the case of a false negative is sufficient 
protection against poor data quality. 

We have already pointed out that the implied knowledge base 
for the decision maker leans heavily on their curriculum 
expertise while presuming little about their numerical competence 
and psychometric understanding. The role of the primary decision 
maker (the teacher) is that of the content specialist. A jnastery 
"".earning model underlines the system and it is presumed that the 
teachers are interested in test results and are willing to build 
in rerouting (additional instruction for a subset of students on 
topics already covered or reteaching topics with generally poor 
mastery). Application of a common curriculum model across 
teachers implies that teacher are willing to work toward a common 
set of objectives for the basic curriculum. 
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The environmetal circumstances are those associated with a 
centralised (or centralizing) district where a decision to 
achieve a commonly agreed-upon set of objectives throughout ir. in 
operation. In this specific setting, l substantial effort was 
made to include teachers extensively in the system development 
and to target reports to assist their instructional monitoring 
activiti within the district-wide framework. It's too early to 
tell (the system is less than 2 years old) whether these 
environmental conditions will lead to a high degree of complience 
or conflict within the teaching force over time. As the system 
becomes connected to the state data bases (in planning at 
present), external influences will perhaps further impact the 
decision makers' use of information. 
Example 2 zz Annual Evaluatior Report 

Exhibits 4 and 5 are taken from an annual report of 
standardized test performance on a single school from a large 
metropolitan school distri > Although the report is unclear 
about the target audience, presumably both the principal and 
teachers (collectively) are the decision makers in this case. 
(The report contained other information besides the test scores.) 

The displays from this example involve a substantial amount 
of information (high IM) although the information is more highly 
summarized than in the earlier example (lower SP). The concern 
for psychometric properties of tae data are also greater (higher 
PQ). 

The most striking distinction from Example 1, however, is 
the substantially greater demands placed on the decision makers' 
knowledge base with respect to quantitative information. While 
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ERIC 



SCHOOLS 



STUDENT ACHIEVEMENT OR C.T.t.S. 
MATCHED PRE-POST GRADE EQUIVALENT SCORES 
SIP STUDENTS ORLT 

SUITESTi TOTAL READINS 



YEAR i 1D7T-80 



GRADE I I PRE I POST I 
LEVEL I N I TEST I TEST I 



ORADE EQUIVALENT VALUES 
♦ 23456 



TEARS CHANOE 
PRE TO POST 



I I NA I 
I 43 I I 



I 



I 

I I t 

1.7 IXXXXXXXXXXXXXXXXX I 



I 



t i 1.7 i 

I 44 I I 

i t I 



IOOOOOOOOOOOOOOOOO I 

I I I 

2.1 txxxxxxxxxxxxxxxxxxxxx 



9.4 



t f 2.2 I 10000000000000000000000 I 
I 34 I I I I I I 

it ! 3.5 ixxxxxxxxxxxxxxxr\xxxxxxxxxxxxx,ixxx 



1.3 



i 43 : 



3.4 ( 10000000000000000000000000000000000 I 

I I I I I I 

t 4.1 IXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 



I 



0.7 



I i 4,0 I lOOOOOOOOOOOOOOOQOOOOOOOOOOOOOObOOOOOOOOO I 

I M ! I I I I I I 1 

I { I 5.4 ixxxxxxxxxxxxxxxxxmxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 



1.4 



I t 4.7 f 100000000000000000000000000000000000000000000000 I 

I 52 I t I I I I I 1 

i t I 5.7 IXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 



1.0 



THE DATA DISPLAYED ABOVE ARE 'HATCHED DATA.' 

DULY FLUENT ENGLISH SPEAKING STUDENTS WHO TOOK BOTH THE 

PAETESTS AND POSTTESTS ARE INCLUDED. 



29 

28 

ERIC M8T COPY AVAILABLE 



Exhibit 5 



DISPLAY OF Cf & l TESf RESULTS FOR GRADE 1 

PRETEST DATA. Spring 1979, CTBS Level A -» POSTTEST DATA, Spring t9 80, CTBS Level B 



COJNT 



IftSTfflfRAM OF TOT4L READING 



1 — ii — — — — — — ri si 5t"°T54 riA r5A 

RAW SCORES 



MEAN PERCENTILE*** 6* 

Mf:0!AN» PERCENTILE* 60 



N. 

MEAN* • I 

MEOIAN* ••••***••***• • l?S*Z2 

STANDARD DEVIATION. •• 18*69 

QUARTILE INFORMATION 

S N 11 : ft ? In ?! : U 5 '« 31 : H ? 18 « : B 

£5 STUDENTS SCORED ABOVE 120 AND HENCE ARE HOT DISPLAYED CN tt€ 
HISTOGRAM ABOVE 



HISTOGRAM CF TOTAL .RATH 



COUNT 
5- 



• 

* t • 



6 

♦ 6 6 

0 * t t t 

* * * * * t t 



6 

t 

• ♦ • 



*********** 



* 



* 



RAW 



-0 l-H~ 

20 25 



— I 
30 



».»»».»........ 

Mf* AS ••••••••••••••••• 

M tOIAN»»»*»********** 

STANDARD DEVI AT I UN. •• 



V C 0 >R F S 
16*68 



MEAN PERCENTILE... 
MEDIAN PERCENTILE* 

oimmi INFORMATION 



*.62 
5.00 



43 

50 



COJNT 
5- 



HIST06RAN C* TOTAL REA0IN6 



** 6 
t t * *♦ * * • *• 



I 1 I — 

10 20 



-I* 
30 



UO 120 



40 50 60 70 80 90 100 

RAW SCORES 

43 MEAN ft.**"*****.* it* 7 

32.09 MEAN PERCENTILE... AX 



N«*« •••#••••••••••••• 



QUART! IE INFORHATtON 



snn:H siasi:55 swh:h sis»:R 



COJNT 
5- 



MISTOORAM OP TOTAL MATH 



*t 
Ml 

****** IMttHt 



# 6 
Ml 6 6 
****** * f 



t 6 
• 6 
***** ♦ * 



I- 



— i — ii- 



-I — i — oi — io — -I- 

13 20 25 30 33 



—1-0—1- 
40 *5 



-I- 
50 



—I 
55 



1 

60 



RAW 



SCORES 



1U 

IMS 



BH MKfttUi:;: is' 8 

MEOIAN PERCENTILE* 58 



Nt ...... 

ME AN... •••••• 

MEOt AN ••••••• 

STANDARD DEVIATION... 

OUARTILE INFORMATION 

inn:n mn:» i»8sm si«k:j? 
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represents 1 student(s); feoly students who were pretested and posttested are included. 
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there is virtually no information about subject matter content, 
the user is presented at least 8 different statistical concepts 
(grade-equivalent, histogram, mean, median, standard deveation, 
mean percentile, median percentile, quartile) plus assorted 
notation ("% in 01") and types of graphic displays. Given the 
literature on teachers and administrator knowledge about testing 
and measurement (e.g. Gullickson, 1984a, 1984b; Mayo, 1967; 
Rudman, et al, 1980) there seems to be a much greater likelihood 
that the targeted decision makers will be unable to use this 
information effectively for educational decision making (Alkin 
and his colleagues (e.g., Alkin et al, 1985; Stecher, Alkin & 
Flesher, 1981) provide some insight into this apparent mismatch 
between information and intended user). 
Example 3— At-A-Glance Reports 

The next set of exhibits are the familiar at-a-glance 
reports from the Systemic Evaluation Project's case study. The 
development and implementation of these reports, and the 
reactions of school personnel to them, are documented and 
analyzed in other project reports (Burstein & Sirotnik, 1984; 
Dorr-Bremme, 1984, 1985; Sirotnik & Burstein, 1985). Here we 
focus on subsets of the questions from the student survey and 
their reporting and use to illustrate how the decision context 
influences specificity decisions with respect to non-achievement 
data. 

Of the 14 columns of information on the final version of the 
student^^-a-glaacji ^report (Exhibit 6), 6 columns (educational 
expectations, academic self-concept, job status, activities, 
homework, like school) are based on information from the student 



!C1I0R: 
iACBZB: 



STODBBIS AT A GLABCE 



CAIIIUIL 0 

FBEPABBD OH 10 CC1 84 



OEERI SICEEBI 
I0BBEB BABE 




J771 
49050 

43177 
44197 
49266 
49288 
11828 
49291 
19069 
49294 
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19301 
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.9168 
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S10AB7. DCBBA 
IHCEFSON, BENE1 1 
M1REB, VICBI S 





r 




B 




s 


G 


C 


B 


B 


A 


O 


O 


O 




i 

JU 


12 


• 


15 

• * 


• 


12 


3 


12 




12 


3 


12 


« 


12 


6 


12 


6 


11 


6 

V 


12 


A 


12 




12 


1 
■ 


12 


2 


12 


6 


12 


7 


M 


4 






12 


1 


12 


1 


12 


6 


12 




12 


5 


l5 


3 
ft 


12 


2 


12 


3 


12 


2 


12 




12 




12 


6 


12 


5 


12 


ft 


12 


5 


12 


i 


12 


5 


12 


6 


12 


1 


12 


8 


12 


4 



E 

I) 

E 
X 

P 
E 

C 
T 

41 

BS 
RS 

Si 
IJ 

2X 
BS 
7 



4X 
4X 

2X 
BS 
2X 
41 

2X 
21 
2X 

41 
41 

21 
21 
21 
4X 

41 

BS 



21 

S? 

2X 
21 
21 
21 



D 
A 

I 

S 

A 

B 

S 



57 
35 

U 

11 

23 
30 
13 

55 
27 
7 
8 
37 
11 
31 
35 
10 
6 

35 
17 
27 
43 
10 
35 
46 
32 
30 
47 

30 
45 
118 
27 
10 
15 
56 
31 
27 
25 



C 
T 

i 

B 
E 
A 
D 



C 
1 
B 

S 

I 
A 
B 

G 



C 
T 
B 

S 

B 
1 

T 
B 



33 1* 3 

52 84 83 

U 54 3 

49 ll 0 

80 88 80 

68 72 68 

eS el .68* 



36 26 

J? ? 

66 84 95 

87 94 83 

29 12 

45 62 

41 20 



4 

10 



€3 64 64 
€6 96 99 



61 
50 
83 



62 40 83 

1 3 m 

8 32 

*? ii 

80 70 60 

23 30 28 

78 50 3 

16 28 45 



99 74 43 



8 0 

47 49 

61 74 

42 17 



7 
1 

6 

73 40 



63 50 
12 12 



52 95 
0 



6 
F 
A 

1.1 



Hi 

2.6 

h 

1.6 

3.0 
1.4 

ill 

3.6 
.4 

,6 

2.1 
1.7 
4.1 

2.4 
2.5 
2.5 

3l2 
2.8 
1.6 
2.9 

I- 6 
2.1 

2.5 
.8 
1.1 

2.9 
3.1 



A 

C 
A 
0 

S 

c 

R 
R 
H 
H 
R 
B 
R 
R 
H 
H 



R 
R 
H 
H 
R 
H 
B 
H 
B 



R 
H 
B 
H 
H 
R 



H 
H 
B 
B 
B 
H 
H 



* BISSIRG 

SCHCC1: CIFEEB RAGBET SCBOOl. 
1-FRIS3CA1 SCIENCE AND TECBBOIOGI 
2*IHIEBMAIIORAI REIATICHSC POLITICAL SCIFRCS 

S«?g?I5 1 i»B H iiIiJiiSil IC |S§iSJ l K §§i ENCES ^heebai ifis 

IIS AES: ROREER Of IOII DAIS ABSENT. 
ES IEST BZSOLTS ARE EEPOBTED IB PEBCENTILE BANK. 
:ad SC: ACAEEHIC SHIP CCBC2PT. b«bigb b«hbeioh i«icb 
:BEH0ER: 4-AIl/HCST CP 1BE TIRE 0-SOHETIHE — SEtEOfl/BEVEB 
IB: M0111IRE(30*) H*BALPTIBB (20-30) P-PABTIIHE (10-20) B-HCRE 

RBRBER OP BXTBACOBBICOIAB AC1ITITIES |1-5). 33 
Uyy^: IIRE OF SCBCCl. 4»12KE 0-BOT 30BI — DIS12KE 



B 
0 
B 
E 
B 
0 
B 
R 

♦ 

0 
♦ 

0 



♦ 
♦ 

0 
♦ 

4 



B ♦ 
B - 
H 0 



♦ 
♦ 
♦ 
♦ 



♦ 
♦ 
♦ 



J 
0 
fi 

P 
F 
H 
fl 
N 
R 
H 
R 



P 
P 
R 
P 
P 
P 
B 
P 
P 

P 
R 
B 

# 

B 
B 
R 
P 
R 
P 



H ♦ P 
H - P 



P 
H 
B 

P 
F 
R 
P 



3 
3 
3 
0 
1 
0 
0 

1 

2 

2 
0 
0 

3 
2 
0 
0 
1 
0 



0 
0 

3 



I 
I 
K 
E 

S 

c 

R 

♦ 
4 

♦ 
♦ 
♦ 

0 

4 



4 
♦ 
♦ 

♦ 

0 



4 

0 



4 

0 

4 



i - 
i - 



BEST COPY AVAILABLE 



survey. *^hile all of the information is reported for each 
student (i.e., no summarization across students), 4 measures 
(educational expectations, job, homework, like school) are more 
or less direct reproductions of students' actual responses while 
the remaining two are summarizations across a set of questions 
answered by the students (Exhibit 7 provides the questions and 
response freTuencies that contribute to the student-at-a-glance 
report ) . 

The rationale behind the selection of these items from the 
student survey to provide on the class roster-like at-a-glance 
report and the different degrees of specificity in the reporting 
are derived from the intended decision context. The pertinent 
attributes of the context are that the reports were intended to 
be distributed to secondary school teachers early in the semester 
to assist in teacher instruction— planning (e.g., Are the amount 
and difficulty of reading material and assignments appropriate 
for the class? Are special motivational efforts necessary with 
this class?). Some persons involved in the design of the report 
(certain teachers from the work group) also believed that the 
information could be used throughout the semester to help 
teachers interpret student performance during the class (e.g., 
Does the student have too many competing committments? Is there a 
history of performance or behavior problems that could account 
for an individual's class performance?). 

The decision context then is one where teachers are expected 
to integrate the information according to their own needs rather 
than rely on externally designed summarizations. The only 
instances of advance summarization are for the measurement of a 
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Exhibit 7 



^Questions) and accompanlng frequencies from Spring 1984) used 1n generating 
Students-At-A-Glance reports for Site A. 



Education Expectations 

8. Actually, I trill probably: 
1 F. Quit school as soon as possible 



23 
30 
40 



G. 
H. 
J. 
K. 



Finish high school. 
Go to trade/technical school or junior college. 
Go to a 4-year college or university. 
Don't know. 



Academic Sel f-Concept (rounded average of 9 questions, scale revlesed on 
questions 15, 21, and 22) 

15. I'm not doing as well as I'd like to In school. 

16. I am a good reader. 

17. I'm proud of my school work. 

18. I'm good at math. 

19. I'm doing the best work that I can. 

20. I an able to do schoolwork at least as well as 
other students. 

21. My grades are not good enough. 

22. I'm always making mistakes In my schoolwork. 

23. I am a good writer. 
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Homework 

102. In general, how often do you do your homework? 
21 F. All of the time 
41 G. Most of the time 
TJ H. Sometimes 
Seldom 



11 
3 



J. 

K. Never 



Job 

4. About how many hours a week do you usually spend working on a Job during th' school year? 

50 F. None. I am not employed during the school year. 

17 G. About 10 hours or less 

18 H. About 15-20 hours 
13 J. About 20 - 30 hours 

3 K. More than 30 hours 
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Activities (number of yes responses to 5 questions) 



39. I participate In sports teams/drill team/flags/cheerleading. 37 

40. I participate In student government. 3 

41. I participate In music, band, drama, or other arts. 17 

42. I participate In honor society. 

43. I participate In school clubs/comnunlly service activities. 



Like School 



Yes No 



60 
88 

72 

19 77 



26 7i 



SR^ fif* 5* mW ' Strongly 

74. I like school. 10 

19 41 14 12 12 • 
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general construct (academic self concept) where the individual 
questions are of limited interest fo* the tearhers and would 
overemphasize minor fluctuations in student responses to items of 
this type and a set of items characterizing specific activities 
where again the general tendency rather than specifics were 
believed to be pertinent. 

The design of the at-a-glance report also reflects the work 
group's perspective with respect to mass of information and 
psychometric quality. While the information provided on all 
class members is quite detailed, the report itself was 
constrained by the amount of information that could be printed on 
a single sheet for each class. Many other pieces of information 
contained in the student survey c available through other 
sources within the district were not included to keep the data 
base for decision making manageable for teachers whose time is 
limited. 

The question of psychometric quality was seemingly resolved 
by accepting the fallible properties of specific questions but 
assuming that decision making would be based on the "patterns" 
evident in the responses. Thus the balance and 
representativeness of the responses was viewed as a means to 
offset overreliance on specific pieces of flawed information. 

The assumptions about the decision maker's knowledge base 
implicit in the report also warrant mention. First, there is 
again only a limited amount of numerical reporting; only test 
scores (percentiles) and g.p.a. involve any statistical 
summarization to speak of and both are reported in units 
generally familiar to teachers. Second, the types of information 
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selected for inclusion in the report suggests that teachers lack 
basic information about the backgrounds (prior performance, 
activities, interests) of the students in their classes. 
Moreover, if teachers had such information, they would use it 
(and know how to use it) in instructional planning and student 
monitoring. 

The evidence from the report use survey and teacher 
interviews (See Dorr-Bremme (1985) and Sirotnik & Burstein (1985) 
for details and extended discussion of these information 
sources.) is that teachers had mixed reactions to the reports. 
While many indicated that they found the report informative, some 
teachers were uninterested in student backgrounds as a basis for 
course planning. Others worried about the expectations that 
knowledge of this information would create. Thus there were clear 
indications of individual differences within the educational 
setting with respect to perceptions of roles and 
responsibilities, and presumably with respect to beliefs about 
teaching. Under such circumstances the system could not be 
expected to be uniformly valuable to all teachers. In fact one 
of the strong signals from the use questionnaire results is that 
secondary teachers are a diverse bunch whose information needs 
could best be served by a customized information system where the 
teacher selects from an available menu of data. 

Environmental conditions were important in this case as 
well. (Dorr-Bremme, 1984,1985). Without a committment from the 
district to participation in the project, without a technically 
proficient, flexible data processing division, and without a 
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prior interest within the school in improving educational 
decision making, the development effort could not have been 
carried out in the first place. Of more particular interest to 
the specificity of information reported from the student survey, 
there was already precedent within the school for maintaining 
highly detailed and personal information about students in a 
computerized system developed for school counselors. In fact 
certain teachers already were accessing these files. Thus there 
uas an atmosphere established where it was not unreasonable to 
ask students personal questions and to share their responses with 
the teaching staff. Later concerns were expressed about the lack 
of anonymity of student data another indication of diversity of 
beliefs and perspectives among the targeted decision makers. 
This mixed reaction also was symptomatic of the environmental 
circumstances under which the system was developed and poses a 
dilemma for its sustains! ility. 

Concluding Comments 
We could continue to present illustrations from other 
information system settings of the applicability of the 
components of our rudimentary model of human decision making to 
the decisions about the specificity of information to include. 
Ideally, one would like to see a taxonomy of decision contexts 
(including types of decisions, types of data, and types of 
decision maker) developed that indicated how the various 
attributes from the decision making model influence the 
appropriate specificity of information as well as other 
characteristics of the information system. Unfortunately, such a 
taxonomy could quickly become encyclopedic. A more modest goal 
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would be to establish the value of considering a few key concepts 
from the psychology of human decision making as a routine part of 
the development of instructional information systems. 

As researchers and practitioners go about "inventing the 
future" of information systems for data-based decision making in 
schools, there will continue to be fits and starts, small successes 
and maybe big failures which will have to be understood and 
explained to sustain real progress. Explanations for 
accomplishments, and lack thereof, in different educational 
settings will continue to rely heavily on the kinds of social 
organizational, political, and historical reasoning that is 
reflected throughout the reports and articles generated by both 
the Systemic Evaluation Project and the Management of 
Instructional Information Systems Project (refer to the reports 
already cited throughout this paper). This paper is intended as 
a small reminder that psychological explanations, especially 
those derived from models of human decision making, are pertinent 
as well. 
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